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Two studies were conducted to assess whether a 


sentence verification technique fcr measuring reading comprehension 
was sensitive to reading difficulty of text. A total cf 96 students 


“in grades four through six réad text passages two grade levels below 


reading level, at reading level, and two grade levels atove reading 
level and then judged whether original, paraphrase, meaning change, 
and distractor sentences were the same or different in meaning from 
sentences in the text. Analysis of the proporticn of correct 
responses and of scores based on a criterion-free measure of response 


accuracy indicated that the technique was sensitive tc text 


difficulty. The sentence verification technique has a numter of 
possible uses in such areas as development of instrtctional 
materials, reading diagnosis, and comprehensicn measurement in 
experimental research. It has an advantage over cther comprehension 
measurement techniques popularly used in standardized testing in that 
it is not influenced by inferential reasoning ability or general 


world knowledge. 
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Abstract 


Two studies are reported which assess whether a sentence verification 
technique for measuring reading comprehension is sensitive to reading 
difficulty of text. Fifth Ga sixth grade students and fourth and sixth 
grade students read text passages two grade tevels below reading level, 
at reading level, and two grades above reading level, and then responded 
to patil, paraphrase, meaning change, and distractor sentences. 
Analysis of proportion correct and d' scores indicated that the tech- 


nique was sensitive to text dtf Frew ty, A number of possible uses and 


advantages of the sentence verification technique are discussed. 
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A Sentence Verification Technique for 


Pe 
Measuring Reading Comprehension 


For the most part, techniques for measuring reading comprehension are 
independent of any theoretical considerations of reading as a psychological 
process. The two most popular techniques involve either having a reader 
examine a text passage and then pape to multiple choice questions, or 
having the reader fill in the blanks in a passage from which,words have 
been deleted (the cloze technique). These techniques have evolved more as 
a function of pragmatism than of theory, and the continued use of the 
techniques is dependent more on psychometric criteria than .on theoretical 
considerations. | 

This paper reports two studies using a technique for measuring reading 

comprehension which is based explicitly on theoretical notions about reading 
comprehension as a psychological process. Briefly, these notions are that — 
language comprehension is a constructive process in which an incoming 
linguistic message, the recipient's prior knowledge, and the environmental 
context in which the message is received all interact to determine the 
nature of the comprehended message (e.g., Bransford & McCarrell, 1974; 
Kintsch, 1974; Kintsch & van Dijk, 1978; Royer & Cunningham, 1978; 
Smith, 1971). Moreover, the nate te representation of this comprehended 
message is cave form which preserves the meaning of the message but not its 
surface structure (Sachs, 1967; 1974). 

Given the above perspective, one way to determine if a sentence has 


been comprehended is to establish whether this operation has been preferred, 


4 
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i.e., whether the sentence is, in fact, represented in memory in'a form which 
preserves the meaning, but not necessarily the surface structure of the S| 
message. We have developed a variation of a sentence verification tech- 

nique which is designed to do precisely this. In our studies students 

read text segments which vary in difficulty, and then are presented with 


four types of test sentences which either are the same or mean the same 


as an original Sentenee or have a meaning different from that of the 
original sentence. The student's task is to distinguish ee these 

two classes of test sentences. If student performance on the task were 

to vary systematically as a function of text sieF ree, we would interpret 
this as one ‘source of evidence pos adele the conclusion that the test is 
measuring peuaree Soniprehend ian | 


e 


y 


Experiment ] 


Method j 

Design, subjects, and reading SAG eras: The intent of the study 
was to determine if aide sentence ee technique could detect 
differences in conprebentian when students were reading material of 


° 


Sey ingalit Fiuatey level: The study was designed such that each subject 
read and wae examined on material which was at a difficulty level well 
below the reading level of the ee reader, at the average reading 
level, and wel] above the average reading level. 

The subjects were 21 fifth graders and 33 sixth graders from a 


small school district in Central Illinois. At our initial meeting with 


the teachers of the two classes we explaimed our technique and showed 


Cy 
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them sample test materials. We then asked the teachers to select text 


samples which were about two grade levels below (in difficulty) the level 


+ 


at which the average child in the class was reading, right at the level 
where the average child was reading; and two grade levels above where, 
the average child was reading. Furthermore, a samples were to be 


ones not. seen by the children before. Thus, we had text samples which 


were approximately at the third, fourth, fifth, sixth,. seventh, and 


eighth grade levels of difficulty. 

| Test materials. Using the text samples identified by the teachers, ° 
we first divided the texts into two segments, each of which was twelve 
sentences jetigi We then developed four test sentences for each of the” 
sentences which sepesied in the original text. These Seubenees were: 
(1) the original sentence exactly as it had appeared in the text, (2) a 
paraphrase of the original sentence in which as many words as possible 
were changed without altering the meaning or the syntactical structure 
of the original sentence, (3) a meaning change sentence in which one word 
tw the original sentence was altered such that ie: cieentas of the sentence 
was changed, and (4) a distractor sentence which was consistent with the 
general theme of the sceuates, but was unrelated to any of the original 
sentences. fi addition, the distractor mantenee was written (intuitively) 
a be of the same length, syntactical structure, and difficulty level as 
the original sentence. Sample items for third and eighth grade materials 


are presented in Table lI. 
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Insert Table | about here 


After developing the four types of sentences for each sentence in the 


original text, we stored the sentences, and the words ''old''.and ''new'! under- 


neath each sentence, on computer tape. A program was then written which 


randomly generated (without replacement) sixteen sentence tests based on 
each of the twelve senténce segments of original text. The single 
restriction to the random rule was that the first eight test sentences in "4g 


v 


a test came from the first six sentences of the original text (this was 


a; ’ 


to control for recall from short term memory). Using this program, we 

| . generated a different test form for each of the students who saviielpubed 

in the study. | | | 

The actual test booklets consisted of the following items: (1) a 
cover sheet which was color coded for easy identification; (2) a hart 
easily understandable contert setting paragraph, the purpose of which was 
to provide a schematic reference for reading the text samples; (3) a text 

“-ssample twelve sentences in length; (4) the sixteen item test based on the 


text sample immediately preceding it; (5) a second context setting paragraph; 


(6) a second text sample at the same level of difficulty as the first: 


Procedure. The first day of the study was devoted to training the 
.students on how to perform on the tests. At the beginning of the training 


period booklets were handed out which were similar to those used in the 


. 


and (7) a second sixteen item test based on the second text sample. 
actual study. We then explained that the purpose of our study was.to try 


’ 
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out a new technique for measuring réading: comprehension and that the 


t ‘ ‘ 


technique involved reading a segment of text and then judging whether 


test sentences were "'old'''or ''new.'' It. was then indicated. that old. 
‘ gs vow % 

sentences were sentences that were the same.or meant the same as. an 
2 : mre : 3 


original sentence, whereas new sentences were either different in-. 


meaning or unrelated to an epigiial Sentenee.. The students were then 
instructed to turn to the first page of their booklets which contained 
a context setting paragraph. After reading this paragraph, the students. 
turned the page to a twelve sentence text sample (estimated to be wet) 
eetaa the average reading level of both classes) we had made up for 

vthe purposes of training. The instrdctions on this page were to read 


» 


the text slowly and carefully twice. After the children had finished 
3 iy 3 te 


reading, we asked them.to turn the page. The page they turned to con- 


tained four’ sentences--one example of each of the- four test sentence 


'* types--based on the first sentence to appear in the original text. At 


the same time we flashed on a screen, (via overhead projector) the same 
four sentences. We then went through ach-oF the four sentences, indi- 
eabihe which of the sentences would be marked old and new (and why), 

and urging the children to duen torthe: attain! text to compare the 
original sentence to the test sentences. ‘After going through the four 
Jebvetipe ie asked for and responded to questions, and then instructed 
the children to turn to the next page. This page contained the test 
sentences for the second sentence appearing in the original text, and as 


( 


the children turned to the page, we projected the sentences onto the 


soy Wad 


‘ 
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screen in-the front of the class. We then asked ‘the chi Idren to shout out 


ra 


3 : i ' ; wo a ¥ i 
how the first sentence would be responded to, The children were then given 
the correct answer and told why it was correct, and we then went on to the 


: Wage | Se ee . * ‘ ‘ 
next sentence. This -procediyre ‘cent inued until we had asked for responses 
~ and questions and given ‘feedback on all the sentences. 


The process described a ve continued until we had examined the test 


sentences baged’ on the first/six sentences in the original text. During 
oo 


Re miss 
_this proeedure it was noted that very few children failed to correctly 


) 


‘respond with the correct ‘classification for each of the test sentences. 
After completing the above described procedure for the six original text 
sentences, we asked the children to turn to a page which contained only 


one of the test sentence types (randomly drawn but the same for each 


child) for each of the remaining six original text sentences. Under each 
of the sentences wer the words ''old'' and '!mew,'' and the children were 
instructed to circle the correct classification for each of the six test 
sentences and to not look back to the original text. After all of the 
children had completed their ratings,we projected the sentences on the 
screen and gave the correct classification for each sentence along with 


‘ 
the reason for the'classification. While this explanation was being 


given, two experimenters walked up and down the ‘aisles giving personal 
attention to any student who had an incorrect answer. 

After a call for, and a response to, general questions, a second 
test booklet was handed out. This booklet contained the seven types of 
material described in the test materials section. The text was based on 


material judged to be at an average level of difficulty. After general 
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instructions to read the context paragraphs, to read the text slowly and 
carefully twice, and to complete the test sentences in the manner practiced, 
the children read through the next two samples, and completed the two sets 
of test sentences at their ne pace. During this period we were avai lable 
to answer any questions which might arise. The purpose of this second test 
booklet was to give the children practice under actual test enaiaie: 

On the next day we conducted the actual study. Each child was given 
an envelope containing a set of general instructions and three test book- 
lets, each of which had a different color cover sheet. These test booklets 
had been made up such that each of the possible combinations of ordering 
the three difficulty levels of text uae -pearty equally represented by gah 
color. After reading the instructions aloud, and having the children read 
along with us, we instructed the children to pull out of thetr envelopes a 
particular colored test booklet. They were then instructed to follow the 
instructions in the booklets, and to raise their hands when they had 
finished. When a child's hand was raised, an experimenter checked the 
booklet to see that all of the test sentences had been responded to, re- 
placed that booklet in the envelope, and gave the child the appropriate 
next booklet. This procedure continued until all of the children had 


completed all three of the test booklets. 


iP 
Dependent variables. The task in this study can be considered as a 


two choice discrimination problem. ylypically, the data of interest in such 
tasks is the proportion of correct classifications, and we will analyze 


. 


proportions in this study. There is a problem, however, with using 


id 
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seepertion correct as.dn index of classification accuracy: Response 
accuracy is confounded with criterion differences. Imagine, for example, 
that one of our students was willing to classify a sentence as ‘'old'' when 
he/she had a slight feeling that the sentence was the same as one appear- 
ing previously. Imagine that another steht classified a sentence as 
old only when he/she was absolutely certain that the sentence was the same 
as an original text sentence. The two students might be identical with 
respect to their ability #6 correctly classify the sentences, but their 
performance would differ markedly because of their criterion differences. 


Fortunately, there is a way to separate response accuracy from the 


criterion for a response. The technique involves the use of signal de- 


tection analysis (e.g., Swets, Tanner, & Birdsall, 1961), in which a cutoff 


parameter (8) and a signal sensitivity (d') parameter are separately 
computed. The d' parameter is a criterion free measure of response 
accuracy (see Healy & Kubovy, 1978 for evidence of this) and will be of 
central interest in this study. A d' score can bis viewed as a Z score 
indicating the likelihood that the individual is responding ack chance. 
Thus, a d' of 3.0 is a very high score indicating performance substantially 


above chance. In contrast, a score near zefo would indicate the individual 


is guessing when ciassi the sente as old or new. ‘ 


Results 


Analyses of variance. Both the proportion of correct Scores and the d' 


scores were analyzed using a 2(grade level, 5 and 6) x 3(low, average, or 
4 
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high difficulty level text material) analysis of variance with grade level. 
a between subjects factor and difficulty ee within subjects factor. 
The means for both dependent variables are presented in Table 2. These 
means are based on 96 responses to test sentences (approximately 32 to 


easy material, 32 to on grade material, etc.) for each individual. 


The analysis of variance for the d' data indicated that difficulty 
level was a significant source of variance, F(1,52) = 5.3, p < .05, and 
the interaction between grade level and difficulty level, F(2,104) = 4.7, 
p < .05, were also significant. 

Of additional interest was performance on the four different types of 
test sentences. Proportion correct on the four types broken down by grade 
and difficulty level is weeecaten in Table 3. An analysis of variance 
on these data indicated that in addition to the previously mentioned sig- 
nificant effects, sentence type was also a reliable source ee wneanees 


F(3,156) = 70.8, p < .01. 


ee ee were 


Insert Table 3 about here 


Correlations. The students who participated in our study had 
recently taken the achievement series from the lowa Tests of Educational 
Development (ITED), and we performed a number of correlational analyses 


using these scores and scores from our sentence varification test. These 


ie 
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correlations should be taken as'merely suggestive since our sample size 
was too small to provide stable correlations and since we uged percentile 


og 


. { : @ 
scores from thé ITED rather than raw scores. Given these cautions, the 


correlations between the ITED reading comprehension subscore and the 


sentence verification overgll proportion correct and overal] d' scores 

A 
ee .73 and .58. Another set of correlations of interest 
were: those between the ShbKe Tend of Educational Ability scores (a. general 
ability subscale of the ITED), ITED reading comprehension scores, and 
the proportion correct and d' scores from the sentence verification test. 
As is generally the case with standardization tests, the relationship 
between the abi tity score and the ITED reading comprehension score was 
relatively high and positive (r = .79). In comparison, the relationship 
between the ability scores and the proport ian correct and d' scores from 
the senbande verification tests were somewhat lower (r = .53 and .41, 


respectively). 


Experiment 2 


When we initially planned our study, we had intended to have a sample 


of fourth graders and a sample of sixth graders so that we could have 


‘children in the two grade levels reading and being tested on some of the 


same .materials. That is, our on grade material for the fourth graders 
AMS. ee 

would be below grade material for the sixth graders, and above grade 

materiat for the fourth graders would be on grade material for the sixth 


graders. : Because of the unavailability of a. group of fourth graders, we 


’ 
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i were not able to carry out this plan in our initial study. Soon after 
completing the first study, however, we were able to find a sample of 


fourth and sixth graders, and we then conducted our study as originally 


intended. 


Method 

Subjects and materials. The subpeete were 20 fourth graders and 22 
sixth graders from a medium sized school district in Central Illinois. 
After developing sevenlatts at about the second grade level for use as 
below grade materials for the fourth graders, we asked the teachers of the 
two classrooms to examine our materials to determine if they would be 
appropriate for their students. The teachers agreed that the materials 
which were approximately at the second, fourth, sixth, and eighth grade 
levels of difficulty would be appropriate for use with their students. 

Design, procedure, dependent variables. With two exceptions, al] 
aspects of the study were identical to those reported for Experiment |. 


wv 
The first aspect that was different was that we reprogrammed our item 


generation routine to construct sixteen-sentence tests to consist of 


exactly four original items, four paraphrase items, four meaning change 


items, and four distractor items. The second change was that rather 


” 


than generating a different test form for each student, we generated 


three test: forms which were used with equal frequency in the study. 
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Results 

Analyses of vabaiee: The ee for both the proportion correct and 
the d' dependent variables are listed in Table 4, These data were analyzed 
in a 2(grade fevetich or 6) x 3(low, average, or high difficulty level) 


analysis of variance with the difficulty factor being a within subjects 


variable. 


The analysis of the proportion correct data indicated that both the 
grade level effect (F = .25) and the interaction effect (F = 1.4) were 
nonsignificant, but that the difficulty effect was reliable, F(2,80) =* % 
18.8, p< .Ol. The analysis of the a scores revealed a similar pattern 
in that the grade level effect (F = 1.5) and interaction (F = 2.0) were 
nonsignificant whereas the effect for difficulty level was significant, & 
F(2,80) = 68.2, p< .Ool. 

As noted earlier, the main reason. for wanting groups of subjects 
two grades apart was to allow a comparison between the nsiips on the 
same material. It is iene to examine the means relevant to this 
comparison for both the proportion correct and the d' dependent variables. 


’ 


Note first that the proportion correct data are disappointing in that the 
i] 


“fourth grader's on grade performance is equal to the performance for the 
sixth graders on the same material. Since this is the same material, 


and since sixth graders presumably read with greater comprehension than 


do fourth graders, it should not be the case that they perform at the a 
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same level. However, now examine the d' data. These scores are much more 


consistent with expectations in that the sixth graders are performing much 


better than the fourth graders when both groups are reading the same 


material. For those readers not familiar with d' scores, it should be 
pointed out that the difference between a mean score of 2.49 for the 


sixth graders and 1.38 for the fourth graders is as great as the differ- 


ence between Z scores of the same magnitude. The implications of these 
comparisons will be discussed later in the paper. 

The final analysis of variance was an analysis usihg proportion 
correct scores which imeluded type of test centente as a variable. This 
analysis indicated that type “of item was a significant source of variance, 
F(3,120) = 40.1, p < .01, with the pattern of means being very similar to 
those reported in Experiment |. 

Correlations. The subjects in the study had recently taken the 
Stanford Achievement Test (SAT) and several of .the scores from this test 
were correlated with the proportion correct and d' scores from our sentence 


verification test. Again, these correlations should be interpreted with 


cautign because of the small sample size and the use of percentile scores 
‘ 
in computing the correlations. The correlations between the SAT reading 


comprehension score and the proportion correct and d' score from the 


sentence verification test were .49 and .50, respectively. 


Discussion 
The primary purpose of the studies reported in-this paper was to 


determine if a sentence verification technique would detect differences 


16 


Measuring Reading Comprehension 
¢ 15 

in performance when subjects read materials of varying difficulty. The 
analyses of variance reported for both studies indicate that the tech- 
nique was sensitive to differences in text difficulty. The analyses of 
both the d' measure and the proportion correct measure found that diffi- 
culty level of rhe materials was a highly significant source of variance. 

5 An important methodological point can.be found in the comparison 
between the proportion correct and the d' dependent variables. In every 
ve the d' measure provided a more powerful (in the sense of detecting 
deviation from the, null hypothesis) analysis than did thet proportion 
correct measure. Furthermore, in the comparison of performance of 
fourth and sixth graders on the same material in Experiment 2, ae pro- 
portion correct measure failed to detect a difference whereas the d' 
measure revealed a very large difference between the two grade levels. 

These comparisons suggest that the criterion for making a response varies 
as a function of the skill level of the reader and the difficulty level 
of the text. This in turn stresses the importance of a criterion-free 
measure of response accuracy. 
In addition to providing a criterion-free measure of response accuracy, 


ry 


d' scores derived from a sentence verification test also possess another 
Z# 


quality which could make them extremely useful as an index of text 


comprehensibi lity. A d' score is an absolute index in the sense that it 
reflects deviation from chance performance independent of the nature of 
the text material. This quality could prove useful in a setting (e.g., 


industrial or military) where instructional materials were being developed 
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with a particular student population in mind. The text developers could 


set a criterion for text comprehensibility (say a d' of 2.0), and 


materials could be run through a test-revision cycle until that criterion 


was reached. | / 
It is also possible that the scores from the senteres verification 
technique have diagnostic utility. The four types of test sentences 
mal have been included because they are sensitive to different hypothesized 
styles of reading. Competent readers should have little difficulty with 


any of the sentence types. Presumably, they have transformed each original 


sentence into a form which preserves the meaning but not the surface | 


structure of the sentence. The reader should then be able to identify 
original ‘and paraphrases as sentences which mean the.same thing as an 
' original text sentence, and to identify meaning change sentences and dis- 
tractors as being different in meaning from the original text sentence. 
In contrast to the able reader, a less capable readef should have 
difficulty with the task. Imagine,,.. for asaneten A word by word reader 
who has difficulty grasping the meaning of the words being read. The 


reader might perform perfectly on the original and distractor sentences 


but perform very poorly on the paraphrase (new words, old meaning) and 
meaning change sentences (old words, new meanings) . 

Another possible type of reader would be one who relied heavily 
on his or her world knowledge while reading, to the point where the 


words were not carefully attended to. This reader might perform wel | 
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(because the words are @ither exactly the same or entirely different), 


i} 


Measuring Reading Comprehension 


\7 


on ‘original and paraphrase sentences, but miss the change in meaning change 
sentences. Moreover, it is possible that this reader would respond ''old!' 
to distractor sentences since they are consistent with the theme of the 
original nseecae, 

Another obvious use for the sentence verification technique is as a 
measure of comprehension in experimental research. Whereas techniques 
like free recall of text have apparent positive qualities, they also have 
several negative qualities. For example, they are difficult to use with 
younger children, and she researcher is frequently concerned that much of 

“the variability in recall among subjects is associated with motivational 
. * and emotional (e.g., shyness) factors. It is likely that several of these 


sources of variability may be lessened with the sentence verification 


. technique. Furthermore, the technique is usuable with a wide variety of 


texts. A is a. 


Another aspect of the sentence verification technique which should, e j 
be noted is that it does not require inferentiol reasoning. A subject. 
can respond to a test sentence by.comparing that sentence to the memorial 
representations of the original sentences. In contrast, the questioning 
techniques and the cloze techniques used on standardized tests are 
heavily fal uenced by ‘pte neat tel reasoning and general world knowledge 


(Pyrczak, 1972; Royer & Cunningham, 1978; Tuinman, 1973-74). This in- 


fluence is, in turn, likely to result in a substantial relationship between 


«~ > . 
reading comprehension test scores and scores in general ability tests 


(cf. the correlations in Experiment 1). Future research will have to 


“, 
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establish the extent to which performance on the sentence verification test 

‘is influenced by general ability factors (e.g., reasoning ability, world 
knowledge). Buti at this point it seems likely that the scores will be é 
less influenced by ability factors than are other popular techniques 

(cf. the aveuiuetons in Experiment 1). 


Even though the results of the two studies reported in this paper are 
encouraging, it is obvious that iiich oaeaven remains to be done. Rules 
for generating the Sartade types of test sentences need to be developed 
rather than doing this on an intuitive basis as we did. Also, it is 
necessary to develop text materials which have known comprehension properties 
rather than simply rely on the judgment of teachers. ne way we plan to 
approach this problem is to develop materials using the text analysis 
procedure developed by Walter Kintsch at the University of Balers (see 
Kintsch & Vipond, 1977; Turner & Seeeass Wee 1). Finally, large scale 
studies Aled to be conducted which would determine the extent to which the 


sentence verification technique is a reliable and valid measure of reading 


comprehension. 
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Sample Sentences for Third and Eighth Grade Materials 


Sentence Type 


Original 


Paraphrase 


Meaning Change 


Distractor 


Third Grade 


One drift was piled 


_ against the garage door. 


One mound blocked the 


entrance to the garage. 


One drift was piled 


against the cellar door. 


If it kept snowing 
the schools would be 


closed next week. 


Eighth Grade 


At length | came to know 
him more thoroughly than 


anyone else. 


After a time | knew him 


better than any other 
4a 


person. 


At length | came to know 
him less thoroughly than 


anyone else. 


The mesas of New Mexico 
are strewn with rocks 


and canyons. 
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Table 2 — 
Mean Proportion Correct and d' as a 


Function of Grade Level and Text Difficulty 


4, 


—_ oe 


Grade 5 Grade 6 
Text Difficulty p(c) d' - pc) di 
Below level ,83 2.5 79 ~1.84 
On level TT 1333 oe . 1.23 sae 
* Above level | .70 {24 .68 1.19 


ve 
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Table 3 


‘ Mean Proportion Correct as a Function of 


Grade, Text Difficulty, and Sentence Type 


‘ ; Grade 5 Grade 6 
Sentence Type ——_ —— 
Below On Above Below On Above 


ng 


Paraphrase -67. .54 56. ‘ .72 + .62 .65 
ning Change - 74.73 .54 . .67. .52 ~ 46 
Distkactor ’ -97 .90 £92 |  ,88 = .86 84 


Original 93.89.80 .88 .86 78 ; | | 
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Table 4 
Mean Proportion Correct and d' asa 


Function of Grade Level and Text Difficulty 


a 
Grade 4, Grade 6 
Text Difficulty ‘P(c) sd! p(c) * d! 
Below level | 88 2.98 84 2.50 
On level 84 1.38 81 1.36 
Above level Wee 186 Ff a es 
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